Methods and systems for movement control of flying devices

ABSTRACT

A method for controlling a movable object includes receiving a user input indicative of a command to adjust a perception of a target while tracking the target, determining a subsequent perception of the target based on the user input, and generating one or more control signals to move the movable object based on the subsequent perception of the target.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Application No.PCT/CN2016/074824, filed on Feb. 29, 2016, the entire contents of whichare incorporated herein by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

The present disclosure relates generally to device movement control and,more particularly, to methods and systems for movement control of flyingdevices.

BACKGROUND

Unmanned aerial vehicles (“UAV”), sometimes referred to as “drones,”include pilotless aircraft of various sizes and configurations that canbe remotely operated by a user and/or programmed for automated flight.UAVs can be used for many purposes and are often used in a wide varietyof personal, commercial, and tactical applications. In manyapplications, UAVs can also be equipped with secondary devices toperform various tasks. For instance, UAVs equipped with imagingequipment, such as cameras, video cameras, etc., can capture images orvideo footage that is difficult, impractical, or simply impossible tocapture otherwise. UAVs equipped with imaging devices find particularuse in the surveillance, national defense, and professional videographyindustries, among others, and are also popular with hobbyists and forrecreational purposes.

Image quality can be affected by multiple aspects of UAV flight control,and it can be difficult to control the movement of a UAV whilesimultaneously operating the imaging equipment attached to the UAV. Forinstance, it can be difficult to effectively operate imaging equipmentattached to a UAV while also precisely controlling the distance of theUAV from the subject or target, the multi-axis spatial orientation ofthe UAV, and the flight stability of the UAV, each of which can affectimage quality. Simultaneous control of a UAV and attached imagingequipment can be even more challenging when the target is in motion orwhile making complicated flight maneuvers during which the user'sspatial frame of reference differs from the spatial frame of referenceof the UAV control system.

SUMMARY

In one aspect, the present disclosure relates to a method of controllinga movable object having a first perspective. The method may includereceiving an input signal from a second perspective, translating theinput signal from the second perspective to the first perspective,causing movement of the movable object based on the translated signal.

In another aspect, the present disclosure relates to a system forcontrolling a movable object having a first perspective. The system mayinclude a controller having one or more processors. The controller maybe configured to receive an input signal from a second perspective,translate the input signal from the second perspective to the firstperspective, and cause movement of the movable object based on thetranslated signal.

In yet another aspect, the present disclosure relates to an unmannedaerial vehicle (UAV) system having a first perspective. The UAV systemmay include one or more propulsion devices and a controller incommunication with the one or more propulsion devices. The controllermay be configured to control the UAV to track a target object, thecontroller comprising one or more processors configured to receive aninput signal from a second perspective, translate the input signal fromthe second perspective to the first perspective, and generate one ormore signals to control the one or more propulsion devices and causemovement of the UAV based on the translated signal.

In yet another aspect, the present disclosure relates to anon-transitory computer-readable medium storing instructions, that, whenexecuted, cause a computer to perform a method of controlling a movableobject having a first perspective, wherein the method includes receivingan input signal from a second perspective, translating the input signalfrom the second perspective to the first perspective, and causingmovement of the movable object based on the translated signal.

In yet another aspect, the present disclosure relates to a method forcontrolling a movable object having a first coordinate system. Themethod may include determining an offset between the first coordinatesystem and a second coordinate system, receiving user input indicativeof a desired movement of the movable object in the second coordinatesystem, and generating control signals in the first coordinate systembased on the user input and the offset between the first and secondcoordinate systems, wherein the control signals are configured to causethe movable object to make a movement in the first coordinate systemaccording to the control signals and the movement in the firstcoordinate system corresponds to the desired movement in the secondcoordinate system.

In yet another aspect, the present disclosure relates to a system forcontrolling a movable object having a first coordinate system. Thesystem may include a controller having one or more processors and beingconfigured to determine an offset between the first coordinate systemand a second coordinate system, receive user input indicative of adesired movement of the movable object in the second coordinate system,generate control signals in the first coordinate system based on theuser input and the offset between the first and second coordinatesystems, wherein the control signals are configured to cause the movableobject to make a movement in the first coordinate system according tothe control signals and the movement in the first coordinate systemcorresponds to the desired movement in the second coordinate system.

In yet another aspect, the present disclosure relates to an unmannedaerial vehicle (UAV) system having a first perspective. The UAV systemmay include one or more propulsion devices and a controller incommunication with the one or more propulsion devices. The controllermay be configured to control the UAV to track a target object, thecontroller comprising one or more processors configured to determine anoffset between the first coordinate system and a second coordinatesystem, receive user input indicative of a desired movement of themovable object in the second coordinate system, and generate controlsignals in the first coordinate system based on the user input and thedifference between the first and second coordinate systems, wherein thecontrol signals are configured to control the one or more propulsiondevice and cause the movable object to make a movement in the firstcoordinate system corresponding to the desired movement in the secondcoordinate system.

In yet another aspect, the present disclosure relates to anon-transitory computer readable medium storing instructions that, whenexecuted, cause a computer to perform a method for controlling a movableobject having a first coordinate system, wherein the method includesdetermining an offset between the first coordinate system and a secondcoordinate system, receiving user input indicative of a desired movementof the movable object in the second coordinate system, and generatingcontrol signals in the first coordinate system based on the user inputand the offset between the first and second coordinate systems, whereinthe control signals are configured to cause the movable object to make amovement in the first coordinate system according to the control signalsand the movement in the first coordinate system corresponds to thedesired movement in the second coordinate system.

In yet another aspect, the present disclosure relates to a method forcontrolling a movable object. The method may include receiving a userinput indicative of a command to adjust a perception of a target whiletracking the target, determining a subsequent perception of the targetbased on the user input, and generating one or more control signals tomove the movable object based on the subsequent perception of thetarget.

In yet another aspect, the present disclosure relates to a system forcontrolling a movable object. The system may include a controller havingone or more processors configured to receive a user input indicative ofa command to adjust a perception of a target while tracking the target,determine a subsequent perception of the target based on the user input,and generate one or more control signals to move the movable objectbased on the subsequent perception of the target.

In yet another aspect, the present disclosure relates to an unmannedaerial vehicle (UAV) system having a first perspective. The UAV systemmay include one or more propulsion devices and a controller incommunication with the one or more propulsion devices and configured tocontrol the UAV to track a target object, the controller comprising oneor more processors configured to receive a user input indicative of acommand to adjust a perception of a target while tracking the target,determine a subsequent perception of the target based on the user input,and generate one or more signals to control the one or more propulsiondevices to move the movable object based on the subsequent perception ofthe target.

In yet another aspect, the present disclosure relates to anon-transitory computer readable medium storing instructions that, whenexecuted, cause a computer to perform a method for controlling a movableobject, wherein the method includes receiving a user input indicative ofa command to adjust a perception of a target while tracking the target,determining a subsequent perception of the target based on the userinput, and generating one or more control signals to move the movableobject based on the subsequent perception of the target.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a movable object with a carrier and a payload and a controlterminal consistent with the present disclosure;

FIGS. 2A and 2B show control terminals that may be used with embodimentsconsistent with the present disclosure;

FIG. 3 shows a controller that may be used with embodiments of thepresent disclosure;

FIGS. 4A-4B illustrates coordinate systems that may be used withembodiments consistent with the present disclosure;

FIGS. 5A-5C show movable objects with payloads consistent withembodiments of the present disclosure;

FIGS. 6A-6C show movable objects with carriers and payloads consistentwith embodiments of the present disclosure;

FIG. 7 shows operations of a movable object consistent with the presentdisclosure;

FIG. 8 shows coordinate systems that may be used with embodiments of thepresent disclosure;

FIGS. 9A and 9B show movable objects, control terminals, and coordinatesystems consistent with embodiments of the present disclosure;

FIG. 10 shows a system for controlling a movable object consistent withthe present disclosure;

FIG. 11 shows a system for controlling a movable object consistent withthe present disclosure;

FIG. 12 shows a system for controlling a movable object that isconsistent with the present disclosure;

FIG. 13 shows a system for controlling a movable object while tracking atarget consistent with the present disclosure;

FIGS. 14A and 14B show systems for controlling a movable object whiletracking a target consistent with the present disclosure;

FIGS. 15A and 15B show systems for controlling a movable object whiletracking a target consistent with the present disclosure; and

FIGS. 16-22 show systems for controlling a movable object while trackinga target consistent with the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

Unmanned aerial vehicles (UAV) are recognized in many industries and inmany situations as useful tools for relieving personnel of theresponsibility for directly performing certain tasks. For instance, UAVshave been used to deliver cargo, conduct surveillance, and collectvarious types of imaging and sensory data (e.g., photo, video,ultrasonic, infrared, etc.) in professional and recreational settings,providing great flexibility and enhancement of human capabilities.

Although they may be “unmanned,” that is, operated without onboardpersonnel, UAVs are often fully or partially operated by off-boardpersonnel who may be responsible for controlling multiple aspects offlight and/or other associated tasks (e.g., controlling cargo, operatingimaging equipment, etc.). In many situations, associated tasks, such asoperating imaging equipment mounted to the UAV, must be performedsimultaneously with flight control, which can be challenging.

For example, in professional photography, filmography, and videography,UAVs may be used to capture footage from stationary and/or movingperspectives that may be otherwise too challenging, impractical, orimpossible for personnel to capture. But the conveniences of UAVs inthese situations do not eliminate the need for UAV operators tocarefully control the imaging equipment to obtain high quality results.Additionally, use of imaging equipment on a UAV requires skilled controlof UAV flight parameters, because image quality can be reduced byinconsistent or unstable flight, such as when flight parameters (e.g.,roll, pitch, yaw, altitude, throttle, relative position or speed withrespect to a target, etc.) are allowed to fluctuate or varysubstantially or otherwise poorly controlled.

As is the case for many types of UAVs, multiple flight parameters can beseparately controlled by the operator via an input device (e.g., aremote control). During complicated flight maneuvers and/or whilecollecting imaging data, even highly skilled UAV operators may finddifficulty in maintaining a sufficient level of control over each flightparameter while attempting to obtain high quality image results. Inthese and other situations that demand a high degree of control of eachflight parameter, UAV operators may wish for reduced complexity ofoverall flight control. Disclosed herein are exemplary embodiments ofmethods, systems, and devices for controlling movable objects, such as aUAV, that may allow operators to control the movable objects withrelative ease and a higher degree of precision. In particular,embodiments described herein allow a user to operate a UAV from theuser's perspective instead of the flight parameters from the UAV'sperspective. For example, as opposed to control a pitch angle of theUAV, the user may simply operate the remote control to send a commandfor the UAV to go up, and methods and systems consistent with thepresent disclosure would convert such an intuitive command into flightcontrol signals or flight parameters that can be directly used to adjustthe flight behavior of the UAV.

FIG. 1 shows an exemplary movable object 10 that may be configured tomove or travel within an environment. Movable object 10 may be anysuitable object, device, mechanism, system, or machine configured totravel on or within a suitable medium (e.g., a surface, air, water,rails, space, underground, etc.). For example, movable object 10 may bean unmanned aerial vehicle (UAV). Although movable object 10 is shownand described herein as a UAV for exemplary purposes of thisdescription, it is understood that other types of movable object (e.g.,wheeled objects, nautical objects, locomotive objects, other aerialobjects, etc.) may also or alternatively be used in embodimentsconsistent with this disclosure. As used herein, the term UAV may referto an aerial device configured to be operated and/or controlledautomatically (e.g., via an electronic control system) and/or manuallyby off-board personnel.

Movable object 10 may include one or more propulsion devices 12 and maybe configured to carry a payload 14. In some embodiments, as shown inFIG. 1, payload 14 may be connected or attached to movable object 10 bya carrier 16, which may allow for one or more degrees of relativemovement between payload 14 and movable object 10. In other embodiments,payload 14 may be mounted directly to movable object 10 without carrier16. Movable object 10 may also include a sensing system 18, acommunication system 20, and a controller 22 in communication with theother components.

Movable object 10 may include one or more (e.g., 1, 2, 3, 3, 4, 5, 10,15, 20, etc.) propulsion devices 12 positioned at various locations (forexample, top, sides, front, rear, and/or bottom of movable object 10)for propelling and steering movable object 10. Propulsion devices 12 maybe devices or systems operable to generate forces for sustainingcontrolled flight. Propulsion devices 12 may share or may eachseparately include or be operatively connected to a power source, suchas a motor (e.g., an electric motor, hydraulic motor, pneumatic motor,etc.), an engine (e.g., an internal combustion engine, a turbine engine,etc.), a battery bank, etc., or combinations thereof. Each propulsiondevice 12 may also include one or more rotary components 24 drivablyconnected to the power source and configured to participate in thegeneration of forces for sustaining controlled flight. For instance,rotary components 24 may include rotors, propellers, blades, nozzles,etc., which may be driven on or by a shaft, axle, wheel, hydraulicsystem, pneumatic system, or other component or system configured totransfer power from the power source. Propulsion devices 12 and/orrotary components 24 may be adjustable (e.g., tiltable) with respect toeach other and/or with respect to movable object 10. Alternatively,propulsion devices 12 and rotary components 24 may have a fixedorientation with respect to each other and/or movable object 10. In someembodiments, each propulsion device 12 may be of the same type. In otherembodiments, propulsion devices 12 may be of multiple different types.In some embodiments, all propulsion devices 12 may be controlled inconcert (e.g., all at the same speed and/or angle). In otherembodiments, one or more propulsion devices may be independentlycontrolled with respect to, e.g., speed and/or angle.

Propulsion devices 12 may be configured to propel movable object 10 inone or more vertical and horizontal directions and to allow movableobject 10 to rotate about one or more axes. That is, propulsion devices12 may be configured to provide lift and/or thrust for creating andmaintaining translational and rotational movements of movable object 10.For instance, propulsion devices 12 may be configured to enable movableobject 10 to achieve and maintain desired altitudes, provide thrust formovement in all directions, and provide for steering of movable object10. In some embodiments, propulsion devices 12 may enable movable object10 to perform vertical takeoffs and landings (i.e., takeoff and landingwithout horizontal thrust). In other embodiments, movable object 10 mayrequire constant minimum horizontal thrust to achieve and sustainflight. Propulsion devices 12 may be configured to enable movement ofmovable object 10 along and/or about multiple axes, as described belowin connection with FIGS. 2, 3A-3C, and 4A-4B.

Payload 14 may include one or more sensory devices 19. Sensory devices19 may include devices for collecting or generating data or information,such as surveying, tracking, and capturing images or video of targets(e.g., objects, landscapes, subjects of photo or video shoots, etc.).Sensory devices 19 may include imaging devices configured to gatheringdata that may be used to generate images. For example, imaging devicesmay include photographic cameras, video cameras, infrared imagingdevices, ultraviolet imaging devices, x-ray devices, ultrasonic imagingdevices, radar devices, etc. Sensory devices 19 may also oralternatively include devices or capturing audio data, such asmicrophones or ultrasound detectors. Sensory devices 19 may also oralternatively include other suitable sensors for capturing visual,audio, and/or electromagnetic signals.

Carrier 16 may include one or more devices configured to hold thepayload 14 and/or allow the payload 14 to be adjusted (e.g., rotated)with respect to movable object 10. For example, carrier 16 may be agimbal. Carrier 16 may be configured to allow payload 14 to be rotatedabout one or more axes, as described below. In some embodiments, carrier16 may be configured to allow 360° of rotation about each axis to allowfor greater control of the perspective of the payload 14. In otherembodiments, carrier 16 may limit the range of rotation of payload 14 toless than 360° (e.g., ≤270°, ≤210°, ≤180, ≤120°, ≤90°, ≤45°, ≤30°, ≤15°etc.), about one or more of its axes.

Carrier 16 may include a frame assembly 26, one or more actuator members28, and one or more carrier sensors 30. Frame assembly 26 may beconfigured to couple the payload 14 to the movable object 10 and, insome embodiments, allow payload 14 to move with respect to movableobject 10. In some embodiments, frame assembly 26 may include one ormore sub-frames or components movable with respect to each other.Actuation members 28 may be configured to drive components of frameassembly relative to each other to provide translational and/orrotational motion of payload 14 with respect to movable object 10. Inother embodiments, actuator members 28 may be configured to directly acton payload 14 to cause motion of payload 14 with respect to frameassembly 26 and movable object 10. Actuator members 28 may be or includesuitable actuators and/or force transmission components. For example,actuator members 28 may include electric motors configured to providelinear or rotation motion to components of frame assembly 26 and/orpayload 14 in conjunction with axles, shafts, rails, belts, chains,gears, and/or other components.

Carrier sensors 30 may include devices configured to measure, sense,detect, or determine state information of carrier 16 and/or payload 14.State information may include positional information (e.g., relativelocation, orientation, attitude, linear displacement, angulardisplacement, etc.), velocity information (e.g., linear velocity,angular velocity, etc.), acceleration information (e.g., linearacceleration, angular acceleration, etc.), and or other informationrelating to movement control of carrier 16 or payload 14 with respect tomovable object 10. Carrier sensors 30 may include one or more types ofsuitable sensors, such as potentiometers, optical sensors, visionssensors, magnetic sensors, motion or rotation sensors (e.g., gyroscopes,accelerometers, inertial sensors, etc.). Carrier sensors 30 may beassociated with or attached to various components of carrier 16, such ascomponents of frame assembly 26 or actuator members 28, or movableobject 10. Carrier sensors 30 may be configured to communicate data andinformation with controller 22 via a wired or wireless connection (e.g.,RFID, Bluetooth, Wi-Fi, radio, cellular, etc.). Data and informationgenerated by carrier sensors 30 and communicated to controller 22 may beused by controller 22 for further processing, such as for determiningstate information of movable object 10 and/or targets.

Carrier 16 may be coupled to movable object 10 via one or more dampingelements configured to reduce or eliminate undesired shock or otherforce transmissions to payload 14 from movable object 10. Dampingelements may be active, passive, or hybrid (i.e., having active andpassive characteristics). Damping elements may be formed of any suitablematerial or combinations of materials, including solids, liquids, andgases. Compressible or deformable materials, such as rubber, springs,gels, foams, and/or other materials may be used as damping elements. Thedamping elements may function to isolate payload 14 from movable object10 and/or dissipate force propagations from movable object 10 to payload14. Damping elements may also include mechanisms or devices configuredto provide damping effects, such as pistons, springs, hydraulics,pneumatics, dashpots, shock absorbers, and/or other devices orcombinations thereof.

Sensing system 18 may include one or more sensors associated with one ormore components or other systems of movable device 10. For instance,sensing system may include sensors for determining positionalinformation, velocity information, and acceleration information relatingto movable object 10 and/or targets. In some embodiments, sensing systemmay also include carrier sensors 30. Components of sensing system 18 maybe configured to generate data and information that may be used (e.g.,processed by controller 22 or another device) to determine additionalinformation about movable object 10, its components, or its targets.Sensing system 18 may include one or more sensors for sensing one ormore aspects of movement of movable object 10. For example, sensingsystem 18 may include sensory devices associated with payload 14 asdiscussed above and/or additional sensory devices, such as a positioningsensor for a positioning system (e.g., GPS, GLONASS, Galileo, Beidou,GAGAN, etc.), motion sensors, inertial sensors (e.g., IMU sensors),proximity sensors, image sensors, etc. Sensing system 18 may alsoinclude sensors or be configured to provide data or information relatingto the surrounding environment, such as weather information (e.g.,temperature, pressure, humidity, etc.), lighting conditions, airconstituents, or nearby obstacles (e.g., objects, structures, people,other vehicles, etc.).

Communication system 20 may be configured to enable communications ofdata, information, commands, and/or other types of signals betweencontroller 22 and off-board entities. Communication system 20 mayinclude one or more components configured to send and/or receivesignals, such as receivers, transmitter, or transceivers that areconfigured to carry out one- or two-way communication. Components ofcommunication system 20 may be configured to communicate with off-boardentities via one or more communication networks, such as radio,cellular, Bluetooth, Wi-Fi, RFID, and/or other types of communicationnetworks usable to transmit signals indicative of data, information,commands, and/or other signals. For example, communication system 20 maybe configured to enable communications between devices for providinginput for controlling movable object 10 during flight, such as a controlterminal (“terminal”) 32.

Terminal 32 may be configured to receive input, such as input from auser (i.e., user input), and communicate signals indicative of the inputto controller 22. Terminal 32 may be configured to receive input andgenerate corresponding signals indicative of one or more types ofinformation, such as control data (e.g., signals) for moving ormanipulating movable device 10 (e.g., via propulsion devices 12),payload 14, and/or carrier 16. Terminal 32 may also be configured toreceive data and information from movable object 10, such as operationaldata relating to, for example, positional data, velocity data,acceleration data, sensory data, and other data and information relatingto movable object 10, its components, and/or its surroundingenvironment. Terminal 32 may be a remote control with physical sticksconfigured to control flight parameters, or a touch screen device, suchas a smartphone or a tablet, with virtual controls for the samepurposes, or an application on a smartphone or a table, or a combinationthereof.

In the example shown in FIGS. 2A and 2B, terminal 32 may includecommunication devices 34 that facilitate communication of informationbetween terminal 32 and other entities, such as movable object 10.Communication devices 34 may include antennae or other devicesconfigured to send or receive signals. Terminal 32 may also include oneor more input devices 36 configured to receive input from a user forcommunication to movable object 10. FIG. 2A shows one exemplaryembodiment of terminal 32 having a plurality of input devices 36configured to receive user inputs indicative of desired movements ofmovable object 10 or its components. It is understood, however, thatother possible embodiments or layouts of terminal may be possible andare within the scope of this disclosure.

Terminal 32 may include input devices, such as input levers 38 and 40,buttons 42, triggers 44, and or other types of input device forreceiving one or more inputs from the user. Each input device ofterminal 32 may be configured to generate an input signal communicableto controller 22 and usable by controller 22 as inputs for processing.In addition to flight control inputs, terminal 32 may be used to receiveuser inputs of other information, such as manual control settings,automated control settings, control assistance settings etc., which maybe received, for example, via buttons 42 and/or triggers 44. It isunderstood that terminal 32 may include other or additional inputdevices, such as buttons, switches, dials, levers, triggers, touch pads,touch screens, soft keys, a mouse, a keyboard, and/or other types ofinput devices.

As shown in FIG. 2B, terminal 32 may also include a display device 46configured to display and/or receive information to and/or from a user.For example, terminal 32 may be configured to receive signals frommovable object 10, which signals may be indicative of information ordata relating to movements of movable object 10 and/or data (e.g.,imaging data) captured using movable object 10 (e.g., in conjunctionwith payload 14). In some embodiments, display device 46 may be amultifunctional display device configured to display information on amultifunctional screen 48 as well as receive user input via themultifunctional screen 48. For example, in one embodiment, displaydevice 46 may be configured to receive one or more user inputs viamultifunctional screen 48. In another embodiment, multifunctional screen48 may constitute a sole input device for receiving user input.

In some embodiments, terminal 32 may be or include an interactivegraphical interface for receiving one or more user inputs. That is,terminal 32 may be a graphical user interface (GUI) and/or include oneor more graphical versions of input devices 36 for receiving user input.Graphical versions of terminal 32 and/or input devices 36 may bedisplayable on a display device (e.g., display device 46) or amultifunctional screen (e.g., multifunctional screen 48) and includegraphical features, such as interactive graphical features (e.g.,graphical buttons, text boxes, dropdown menus, interactive images,etc.). For example, in one embodiment, terminal 32 may include graphicalrepresentations of input levers 38 and 40, buttons 42, and triggers 44,which may be displayed on and configured to receive user input viamultifunctional screen 48. In some embodiments, terminal 32 may beconfigured to receive all user inputs via graphical input devices, suchas graphical versions of input devices 36. Terminal 32 may be configuredto generate graphical versions of input devices 36 in conjunction with acomputer application (e.g., an “app”) to provide an interactiveinterface on the display device or multifunctional screen of anysuitable electronic device (e.g., a cellular phone, a tablet, etc.) forreceiving user inputs.

In some embodiments, display device 46 may be an integral component ofterminal 32. That is, display device 46 may be attached or fixed toterminal 32. In other embodiments, display device may be connectable to(and dis-connectable from) terminal 32. That is terminal 32 may beconfigured to be electronically connectable to display device 46 (e.g.,via a connection port or a wireless communication link) and/or otherwiseconnectable to terminal 32 via a mounting device 50, such as by aclamping, clipping, clasping, hooking, adhering, or other type ofmounting device.

In some embodiments, terminal 32 may be configured to communicate withelectronic devices configurable for controlling movement and/or otheroperational aspects of movable object 10. For example, display device 46may be a display component of an electronic device, such as a cellularphone, a tablet, a personal digital assistant, a laptop computer, orother device. In this way, users may be able to incorporate thefunctionality of other electronic devices into aspects of controllingmovable object 10, which may allow for more flexible and adaptablecontrol schemes to be used. For example, terminal 32 may be configuredto communicate with electronic devices having a memory and at least oneprocessor, which control devices may then be used to provide user inputvia input devices associated with the electronic device (e.g., amultifunctional display, buttons, stored apps, web-based applications,etc.). Communication between terminal 32 and electronic devices may alsobe configured to allow for software update packages and/or otherinformation to be received and then communicated to controller 22 (e.g.,via communication system 20).

It is noted that other control conventions that relate inputs receivedvia terminal 32 to desired or actual movements of movable device 10 maybe used, if desired.

As shown in FIG. 3, controller 22 may include one or more components,for example, a memory 52 and at least one processor 54. Memory 52 may beor include non-transitory computer readable medium and can include oneor more memory units of non-transitory computer-readable medium.Non-transitory computer-readable medium of memory 52 may be or includeany type of disk including floppy disks, optical discs, DVD, CD-ROMs,microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data. Memory units mayinclude permanent and/or removable portions of non-transitorycomputer-readable medium (e.g., removable media or external storage,such as an SD card, RAM, etc.).

Information and data from sensing system 18 may be communicated to andstored in non-transitory computer-readable medium of memory 52.Non-transitory computer-readable medium associated with memory 52 mayalso be configured to store logic, code and/or program instructionsexecutable by processor 54 to perform any suitable embodiment of themethods described herein. For example, non-transitory computer-readablemedium associated with memory 52 may be configured to storecomputer-readable instructions that, when executed by processor 54,cause the processor to perform a method comprising one or more steps.The method performed by the processor based on the instructions storedin the non-transitory computer readable medium may involve processinginputs, such as inputs of data or information stored in thenon-transitory computer-readable medium of memory 52, inputs receivedfrom terminal 32, inputs received from sensing system 18 (e.g., receiveddirectly from sensing system or retrieved from memory), and/or otherinputs received via communication system 20. The non-transitorycomputer-readable medium may be configured to store sensing data fromthe sensing module to be processed by the processing unit. In someembodiments, the non-transitory computer-readable medium can be used tostore the processing results produced by the processing unit.

Processor 54 may include or more processors and may embody aprogrammable processor (e.g., a central processing unit (CPU). Processor54 may be operatively coupled to memory 52 or another memory deviceconfigured to store programs or instructions executable by processor 54for performing one or more method steps. It is noted that method stepsdescribed herein may be stored in memory 52 and configured to be carriedout by processor 54 to cause the method steps to be carried out by theprocessor 54.

In some embodiments, processor 54 may include and/or alternatively beoperatively coupled to one or more control modules, such as atranslation module 56 and tracking control module 58, which will beexplained in greater detail below. Translation module 56 may beconfigured to control methods of translating information, such asinputs, command, and other signals, from one perspective (e.g., aperspective of the user, a perspective of the movable object 10, etc.)to another perspective (e.g., another of the perspective of the user,the movable object 10, or another perspective). Tracking control module58 may be configured to help control propulsion devices 12 of movableobject 10 to adjust the spatial disposition, velocity, and/oracceleration of the movable object 10 with respect to six degrees offreedom (e.g., there translational directions along its coordinate axesand three rotational directions about its coordinate axes). Translationmodule 56 and tracking control model 58 may be implemented in softwarefor execution on processor 54, as illustrated in FIG. 6, or may beimplemented in hardware or software components separate from processor54 (not shown in the figure).

Processor 54 can be operatively coupled to the communication system 20and be configured to transmit and/or receive data from one or moreexternal devices (e.g., terminal 32, display device 46, or other remotecontroller). Any suitable means of communication can be used to transferdata and information to or from controller 22, such as wiredcommunication or wireless communication. For example, communicationsystem 20 can utilize one or more of local area networks (LAN), widearea networks (WAN), infrared, radio, Wi-Fi, point-to-point (P2P)networks, telecommunication networks, cloud communication, and the like.Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication system20 can transmit and/or receive one or more of sensing data from thesensing system 18, processing results produced by the processor 54,predetermined control data, user commands from terminal 32 or a remotecontroller, and the like.

The components of controller 22 can be arranged in any suitableconfiguration. For example, one or more of the components of thecontroller 22 can be located on the movable object 10, carrier 16,payload 14, terminal 32, sensing system 18, or an additional externaldevice in communication with one or more of the above. In someembodiments, one or more processors or memory devices can be situated atdifferent locations, such as on the movable object 10, carrier 16,payload 14, terminal 32, sensing system 18, additional external devicein communication with one or more of the above, or suitable combinationsthereof, such that any suitable aspect of the processing and/or memoryfunctions performed by the system can occur at one or more of theaforementioned locations.

The flight behavior of movable object 10 may be understood andcontrolled in a defined coordinate system. For example, FIG. 4Aillustrates a local coordinate system defined with respect to themovable object 10 for describing movements from the perspective ofmovable object 10. The local coordinate system may include three axes,such as an X-axis (e.g., a first horizontal axis), a Y-axis (e.g., asecond horizontal axis), and a Z-axis (e.g., a vertical axis). Movementsof movable object 10 may include roll, pitch, yaw, horizontaltranslations (e.g., left, right, forward, backward, etc.), verticaltranslation (e.g., height or altitude), horizontal speeds, verticalspeed, rotational speeds (e.g., angular, radial, tangential, axial,etc.), and accelerations (e.g., horizontal, vertical, rotational, etc.).Each axis of the local coordinate system may be associated with one ormore particular position or movement parameters that may be changed oradjusted during flight to facilitate effective control of movable object10.

For instance, in the exemplary local coordinate system of FIG. 4A, eachof the X-axis, Y-axis, and Z-axis may be associated with translationalmovements and linear displacements along or in the direction of therespective axis, as well as rotational movements and angulardisplacements about the respective axis. In the example of FIG. 4A, theX-axis may also be referred to as a pitch axis, about which movableobject 10 may undergo pitch rotational movements (e.g., movementstending to tilt one of a front or rear of movable object 10 upward whiletilting the other downward) and along which movable object 10 mayundergo side-to-side (e.g., left or right) translational movements. TheY-axis may be referred to as a roll axis, about which the movable object10 may undergo roll rotational movements (i.e., movements tending totilt one of a left or right side of movable object 10 upward whiletilting the other side downward) and along which movable object 10 mayundergo forward and backward translational movements. The Z-axis may bereferred to as a yaw axis, about which the movable object 10 may undergoyaw rotational movements (i.e., rotational movements on or parallel witha plane defined by the X- and Y-axes) and along which movable object 10may undergo up and down (i.e., vertical or altitudinal) translationalmovements. A person of ordinary skill in the art would appreciate thatmore or fewer axes, or different axis conventions may be used. It isalso noted that directional and planar descriptions (e.g., side-to-side,back and forth, up and down, horizontal, vertical, etc.) are used merelyfor purposes of example and clarification and are not limiting.

Conventionally, control of the flight of movable object 10 requirescontrol of flight parameters in movable object 10's local coordinatesystem, such as speed along a certain axis, pitch amount and direction,yaw amount and direction, etc. Terminal 32 may include controlmechanisms for the user to control the flight parameters with respect tothe local coordinate system.

For example, referring to FIGS. 2A and 2B, first input lever 38 onterminal 32 may be configured to receive one or more user inputsindicative of one or more aspects of controlling movement of movableobject 10. Aspects of controlling movement of movable object 10 mayinclude flight control aspects and payload control aspects. Flightcontrol aspects may include control of one or more aspects of flightachievable by movable object. For instance, flight control aspects mayinclude desired translational movements, desired rotational movements,desired speeds, and desired accelerations of movable device 10. Desiredtranslational movements may include desired vertical or horizontalmovements with respect to the perspective of the user, the perspectiveof movable object 10, a reference perspective, or a differentperspective. Desired rotational movements may include desired rotationalmovements of movable object 10 about one or more axes of a coordinatesystem associated with a perspective (e.g., the perspective of the user,movable object 10, reference perspective, etc.) or with respect toanother object, such as a target. That is, in addition to rotating aboutan axis of a coordinate system associated with a perspective, desiredrotational movements may refer to movements about a reference pointassociated with a still or moving object or target.

In one embodiment, first input lever 38 may be configured to receive oneor more inputs corresponding to one or more desired translational orrotational movements of movable object 10. For example, first inputlever 38 may be a multi-axis control device, such as a control stick,configured to be displaced in a plurality of directions, each directioncorresponding to a type and sign (e.g., positive, negative, forward,backward, etc.) of a command indicative of a desired movement. Theamount of displacement of first input lever 38 from a neutral positionmay be indicative of an extent or magnitude of the corresponding desiredmovement. For example, in one embodiment, first input lever 38 may bemovable (e.g., tiltable) in a forward direction, backward direction,left direction, and a right direction from the perspective of the user.Displacement in the forward and backward directions may correspond todesired movements along a first axis of a coordinate system forperceiving, describing, or defining the movements of movable device 10.For instance, displacement in the forward direction may be indicative ofdesired linear movement in a forward direction, while displacement inthe backward direction may be indicative of desired linear movement in abackward (i.e., opposite) direction. Displacement of first input lever38 in the forward direction may also or alternatively correspond to adesired rotational movement of movable device 10 about its pitch axis.For instance, displacement in the forward direction may be indicative ofa desired rotational movement in a first rotational direction, whiledisplacement in the backward direction may be indicative of a desiredrotational movement in a second (i.e., opposite) rotational direction,both about the pitch axis of movable object 10. The amount or degree ofdisplacement of first input lever 38 in the forward or backwarddirection may be indicative of a desired linear speed or accelerationalong the first axis and/or a desired rotational speed or accelerationabout the first axis. A person of ordinary skill in the art wouldappreciate that other control conventions may be used, and that controlfunctions may be divided among more or a different number of inputdevices.

Displacement of first input lever 38 in the left and right (i.e.,side-to-side) directions may correspond to desired movements along asecond axis of a coordinate system for perceiving, describing, ordefining the movements of movable device 10. For instance, displacementin the right (i.e., first side) direction may be indicative of desiredlinear movement in a right (i.e., first side) direction, whiledisplacement in the left direction may be indicative of desired linearmovement in a left (i.e., opposite or second side) direction.Displacement of first input lever 38 in the right and left directionsmay also or alternatively correspond to a desired rotational movement ofmovable device 10 about its roll axis. For instance, displacement in theright direction may be indicative of a desired rotational movement in afirst rotational direction, while displacement in the left direction maybe indicative of a desired rotational movement in a second (i.e.,opposite) rotational direction, both about the roll axis of movableobject 10. The amount or degree of displacement of first input lever 38in the right or left directions may be indicative of a desired linearspeed or acceleration along the second axis and/or a desired rotationalspeed or acceleration about the second axis.

Second input lever 40 on terminal 32 may be configured to receive one ormore user inputs indicative of one or more aspects of controllingmovement of movable object 10. In one embodiment, second input lever 40may be configured to receive one or more inputs corresponding to one ormore desired translational or rotational movements of movable object 10.For example, second input lever 40 may be a multi-axis control device,such as a control stick, configured to be displaced in a plurality ofdirections, each direction corresponding to a type and sign (e.g.,positive, negative, forward, backward, etc.) of a command indicative ofa desired movement. The amount of displacement of second input lever 40from a neutral position may be indicative of an extent or magnitude ofthe corresponding desired movement. For example, in one embodiment,second input lever 40 may be movable (e.g., tiltable) in a forwarddirection, backward direction, left direction, and a right directionfrom the perspective of the user. Displacement in the forward andbackward directions may correspond to desired movements along a thirdaxis of a coordinate system for perceiving, describing, or defining themovements of movable device 10. For instance, displacement in theforward direction may be indicative of desired linear movement in anupward direction, while displacement in the backward direction may beindicative of desired linear movement in a downward (i.e., opposite)direction. Displacement of second input lever 40 in the forward orbackward direction may also or alternatively correspond to a desiredpower output level of propulsion devices 12. For example, displacementof second input lever 40 in the forward or backward direction maycorrespond to a desired throttle increase or decrease, respectively.That is, displacement in the forward direction may be indicative of botha desired throttle increase and a desired corresponding height oraltitude increase, while displacement in the backward direction may beindicative of a desired throttle decrease and a corresponding height oraltitude decrease. The amount or degree of displacement of second inputlever 40 in the forward or backward direction may be indicative of adesired linear speed or acceleration along the third axis.

Displacement of second input lever 40 in the left and right (i.e.,side-to-side) directions may correspond to desired rotational movementsabout the third axis of a coordinate system for perceiving, describing,or defining the movements of movable device 10. For instance,displacement of second input lever 40 in the right direction may beindicative of a desired rotational movement in a first rotationaldirection about the third axis, while displacement in the left directionmay be indicative of a desired rotational movement in a second (i.e.,opposite) rotational direction, both about the yaw axis of movableobject 10. The amount or degree of displacement of second input lever 40in the right or left directions may be indicative of a desiredrotational speed or acceleration about the third axis.

As mentioned above, experience and skills are required for a user tocontrol the various aspects of movable object 10's movements,particularly so during complicated flight maneuvers and/or when the userhas to control the operations of attached equipment such as a camera,not only because it may be counterintuitive to think of such flightparameters as pitch, yaw, roll, but also because the perspectives of theuser and movable object 10 are often independent and different from eachother. For example, when the user is viewing movable object 10 in adirection not aligned with the X-axis of its local coordinate system,the user often has to make a great effort to mentally adjust to theperspective of movable object 10 or physically move or rotate his/herbody and head to align with the perspective of movable object 10 toachieve effective control.

Consistent with embodiments of the present disclosure, user control maybe provided, received, and interpreted from the user's perspective, andsubsequently converted into flight control signals from movable object10's perspective, such as its local coordinate system shown in FIG. 4A.That way, the user may issue commands in his/her own perspective, oftenintuitive commands such as left turn, right turn, go up, move closer toa target, move faster, etc., which may then be translated into commandsunderstood by movable object 10, such as pitch, yaw, roll, throttle,etc. Terminal 32 may be configured to receive user input correspondingto such commands. Terminal 32 may convert or translate user input in theuser's perspective into signals in the perspective of movable object 10and transmit the same to movable object 10. Alternatively, terminal 32may transmit user input in the user's perspective to movable object 10,which then converts or translates the user input into signals in theperspective of movable object 10 before applying the same to the flightparameters.

In some embodiments, terminal 32 and controller 22 may be configured toswitch between a first mode in which user inputs received via terminal32 directly correspond to movements of movable object 10 in theperspective of movable object 10, as discussed above in connection withFIGS. 2A-B and 4A, and a second mode in which user inputs received viaterminal 32 correspond to movements of movable object 10 in the use'sperspective. For instance, terminal 32 may include a button, switch,knob, a touchscreen icon, or some other type of input or input deviceconfigured to receive a user input indicative of a user selection toenter the first or second mode. A pattern of lever operations may alsobe predefined to effect the switch between modes or selection of modes.Alternatively, controller 22 may assume a default mode, such as eitherof the first mode or the second mode upon being energized or uponreceipt of initial user inputs indicative of flight commands. When inthe first mode, controller 22 may be configured to receive user inputsindicative of flight parameters (e.g., roll, pitch, yaw, throttle, etc.)in the user's perspective and generate commands to movable object 10indicative of corresponding flight parameters in the perspective ofmovable object 10 without translation. That is, in the first mode, userinputs may be indicative of adjustments to flight parameters of movableobject 10 in the perspective of movable object 10. When in the secondmode, controller 22 may be configured to receive user inputs indicativeof flight parameters (e.g., roll, pitch, yaw, throttle, etc.) to causedesired movement of movable object 10 in the user's perspective andgenerate translated commands to movable object 10 indicative of flightparameters in the perspective of movable object 10 that cause movementsof movable object 10 that correspond to the desired movements of movableobject 10 from the user's perspective. That is, in the second mode, userinputs may be indicative of adjustments to flight parameters movableobject 10 in the user's perspective.

As used herein, the term “perspective” may refer to a convention withrespect to which the position and movements of movable object 10 andother objects may be determined, observed, measured or quantified,controlled, or commanded. For instance, a perspective of movable object10 may be or include the exemplary local coordinate system shown in FIG.4A. The local coordinate system may allow movements of movable object 10and movement commands to be perceived or defined from the perspective ofmovable object 10 (i.e., with respect to the local coordinate system).For example, the local coordinate system may be defined or establishedfrom a fixed point on movable object 10 (for example the center point ofmovable object 10) and in defined directions (for example with the X andY axes pointing to middle points between rotary components 24 and the Zaxis perpendicular to the X and Y axes), allowing movements and commandsfor movements of movable object 10 to be perceived, understood,characterized, or defined with respect to the fixed point and the localcoordinate system. In this way, movements and commands for movements ofmovable object 10 may be definable and understandable independently ofother coordinate systems, such as a reference coordinate system (e.g.,global coordinate system, universal coordinate system, a positioningsystem coordinate system, etc.), as means for directly detectingmovements of movable object 10 with respect to other perspectives orcoordinate systems may not be available in all situations.

Movements and positions of other objects or features may also bedescribed from the perspective of movable object 10 with respect to thelocal coordinate system. For example, relative positions (e.g.,distances, orientations, etc.) and relative movements (e.g., speeds,accelerations, rotations, etc.) of other objects, such as personnel,landscape features, vehicles, buildings, etc., may be described from theperspective of movable object 10 with respect to the local coordinatesystem. In this way, the local coordinate system may be used to generatecontrol signals for commanding movable object 10 to achieve desiredchanges to the relative position, speed, and/or acceleration of movableobject 10 with respect to other objects.

Commands for moving movable object 10 may be received from any suitablesource, such as a user, which may have its own perspective of theposition and movements of movable object 10 with respect to itself orother objects. Alternatively, commands for moving movable object 10 maybe received from a reference perspective associated with the user oranother command source.

For example, and with reference to FIG. 4B, other perspectives relevantto movement control of movable object 10 may include a perspective of anoperator or user, a perspective of payload 14, and or otherperspectives. For instance, as shown in FIG. 4B, the user's perspectivemay be or include a user coordinate system. The user coordinate systemmay be a 3-axis coordinate system (e.g., X′-axis, Y′-axis, Z′-axis) andsimilar to the local coordinate system but defined instead from a pointof view of the user, which may allow positions and movements (andcommands for changing the positions and movements) of movable object 10to be perceived, understood, characterized, or defined with respect tothe user's perspective. In this way, movements and commands formovements of movable object 10 may be definable and understandableindependently of other coordinate systems and perspectives. The point ofview and perspective of the user and the user coordinate system may bedefined or established with respect to an aspect of the user, such as anoperator station (e.g., a control terminal, an input device forreceiving user input, an operator's seat, a remote control used by theoperator, etc.) or a fixed point on the user.

Other perspective may include a reference perspective that is associatedwith a reference coordinate system. The reference perspective may beassociated with personnel, equipment, or other objects that participatein controlling movement of movable object 10. For example, the referenceperspective may be the perspective of a management or control facility,the perspective of a server or computer used to carry out controlprocesses, or the perspective of a sensory device used in a movementcontrol process, such as a positioning system or positioning device(e.g., a GPS device). It is understood that other reference perspectivesmay be possible and are not limited to the above-mentioned perspectives.

As also shown in FIG. 4B, other perspectives may include the perspectiveof payload 14, carrier 16, or sensor devices 19 attached thereto, which,for sake of simplicity and convenience of this disclosure, may bereferred to as a payload perspective. The payload perspective may beassociated with a payload coordinate system. The payload coordinatesystem may have 3 axes (e.g., X_(vis), Y_(vis), and Z_(vis)) and besimilar to the local coordinate system but defined instead from a pointof view of payload 14, carrier 16, or sensory devices 19 attachedthereto, which may allow positions and movements (and commands forchanging the positions and movements) of movable object 10 to beperceived, understood, characterized, or defined with respect to thepayload perspective.

The payload perspective may differ (i.e., be offset from) theperspective of moveable object 10 any time payload 14 or its components(e.g., carrier 16 or sensory devices 19) move with respect to movableobject 10. Referring to FIGS. 5A-5C, when payload 14 is directly fixedto movable object 10 (i.e., without carrier 16 or with carrier 16 in afixed orientation), the perspective of payload 14 (e.g., perspective inthe payload perspective coordinate system defined with three axesX_(vis), Y_(vis), and Z_(vis)) may be the same as the perspective ofmovable object 10. Alternatively, payload 14 may be directly mounted tomovable object 10 such that a known or determinable (e.g., measurable,calculable, etc.) fixed offset between the perspective of payload 14 andthe perspective of movable object 10 may exist, which may allow for asimple relationship between movements or commands for movements ofmovable object 10 and movements perceived from payload 14. For example,as shown in FIG. 5A, payload 14 may rotate about a first axis withmovable object 10 when movable object 10 is tilted during flight.Similarly, as shown in FIG. 5B, payload 14 may rotate with movableobject 10 about a second axis as movable object 10 is rotated duringflight. And as shown in FIG. 5C, payload 14 may rotate with movableobject 10 as movable object 10 is tilted during flight. As used herein,an offset in the context of differences between a first perspective(e.g., having a first coordinate system) and a second perspective (e.g.,having a second coordinate system) may refer to an angular difference(i.e., an angle or angular displacement) between at least one aspect thefirst perspective (e.g., at least a first axis of the first coordinatesystem) and at least one aspect of the second perspective (e.g., atleast a first axis of the second coordinate system).

When payload 14 is attached or connected to movable object 10 viaadjustable carrier 16, the perspective of payload 14 may vary withrespect to the perspective of movable object 10. For example, withreference to FIGS. 6A-6C, carrier 16 may permit payload 14 to rotateabout one or more axes (e.g., X_(vis), Y_(vis), and/or Z_(vis) axis) inthe payload coordinate system. For example, as shown in FIG. 6A, payload14 may rotate independently about a first axis, such as X_(vis)regardless of the orientation of movable object 10. Similarly, as shownin FIG. 6B, payload 14 may rotate independently about a second axis,such as Z_(vis), regardless of the orientation of movable object 10. Andas shown in FIG. 6C, payload 14 may rotate independently about a thirdaxis, such as Y_(vis), regardless of the orientation of movable object10. In this configuration, movement of payload 14 about one or more ofits axes and/or in combination with movement of movable object 10 aboutits axes may allow users to adjust the perspective of payload 14 withhigh precision. The ability to precisely control the perspective ofpayload 14 may be particularly important when movable object 10 isconfigured to carry optical equipment (e.g., photographic cameras, videocameras, sensors, etc.) to capture imaging data, such as in aprofessional photo or video shoot.

With reference to FIG. 7, the user may be positioned at a first locationwhere the user may be able to control and observe movements of themovable object 10. Movable object 10 may be tracking and capturing avideo or photo of a target while circling the target. Most of the time,the perspective of movable object 10 (i.e., the local coordinate systemof movable object 10) is offset from the perspective of the user (i.e.,the user coordinate system). A command entered by the user to indicate,for example, a forward translational movement causes a forwardtranslational movement of movable object 10 in the perspective ofmovable object 10 so that movable object 10 moves closer to the target,but such movement does not necessarily appear to be a forwardtranslational movement in the user's perspective. In other words, whenthere is an offset between the perspectives of the user and the movableobject 10, directional commands generated by the user to cause movementsin certain directions in the perspective of movable object 10 in fact,counterintuitively, appear to cause movements in different directions asperceived by the user than what the user may have expected.

FIGS. 8 and 9A-9B help illustrate the inconsistency between the user'sexpectation and perception caused by the offset between coordinatesystems. FIG. 8 shows three coordinate systems, each having an offsetfrom the others. When all axes are aligned, perception from onecoordinate system will be the same as from the other, which can allowcommands generated in a first coordinate system to produce the samemovements of movable object 10 in other coordinate systems as would beperceived from the first coordinate system. However, when the coordinatesystems are offset from each other, a command to cause movement along anaxis of a first coordinate system (e.g., in the positive X direction ofthe user coordinate system) may generate movement in a differentdirection with respect to another coordinate system (e.g., the positiveX direction in the local coordinate system). During fast pace and/orcomplicated flight maneuvers, perception of movement from theperspective of the user can be difficult to reconcile with the resultantmovement from the perspective of the movable object.

As shown in FIG. 9A, when the user coordinate system and the localcoordinate system of movable object 10 are aligned, resultant movementin the coordinate system of movable object 10 matches the desiredmovement in the user coordinate system. But when the user coordinatesystem and the local coordinate system of movable object 10 are offset,the user's commands may cause movable object 10 to move in a differentdirection than what the user desired in his/her perspective. Thedifference between the desired movement and the resultant movement asperceived varies with the degree of offset between the user'sperspective and the perspective of movable object 10.

Consistent with embodiments of the present disclosure, to help usersachieve desired movements of movable object 10 during flight, especiallyin situations where the user's perspective is offset from theperspective of the movable object, controller 22 may be configured totranslate user inputs from the user's perspective into the perspectiveof movable object 10. For example, referring to FIG. 9B, a signal 80from terminal 32, such as a signal for movable object 10 to move alongor rotate about the yaw axis is assumed to be from the user'sperspective, i.e., a signal along the Z′ axis in the user coordinatesystem. Controller 22 translates signal 80 into the perspective ofmovable object 10 and generates three components 80-X, 80-Y, and 80-Zalong the X, Y, and Z axes of the local coordinate system, respectively.The three components may then be used to command or cause correspondingmovements along the X, Y, and Z axes.

By assuming user input on terminal 32 to be from the user's perspectiveand translating the user input into the perspective of movable object10, controller 22 and the system consistent with embodiments of thepresent disclosure allow users to disregard any offset between the twoperspectives and simply indicate a desired movement of movable object 10from his/her own perspective. For example, if a user wants movableobject 10 to move up and forward, i.e., up and away, from his/herperspective, the user may simply tilt a pitch stick on terminal 32.Traditionally, this user input on terminal 32 would be received bymovable object 10 as a command to move in a pitch direction in theperspective of movable object 10. With the translation by controller 22,movable object 10 would instead move up and forward from the user'sperspective, as if movable object 10 has now assumed the user'sperspective, when the movement in fact may encompass movements in someor all of yaw, roll, and pitch directions in the perspective of movableobject 10.

Controller 22 may perform the translation between the two perspectivesthrough matrix transformation, e.g., by constructing a matrixrepresentation of the user input (i.e., in terms of the user coordinatesystem) and transforming the matrix into a command matrix representationof the user input (i.e., in terms of the local coordinate system) basedon the offset between the user's perspective and the perspective ofmovable object 10.

The difference or offset between the user perspective and theperspective of the movable object may be determined in a number of ways.In one example, as shown in FIG. 10, controller 22 may be configured todetermine a direction of at least one axis of the user coordinate systemand a direction of at least one axis of the local coordinate systembased on one or more inputs indicative of a direction of each respectiveaxis with respect to a reference direction. For instance, controller 22may be configured to receive a first directional signal indicative ofthe direction of the first axis of the local coordinate system of themovable object. Controller 22 may receive the first directional signalfrom a first directional indicator, such as a compass, a positioningdevice, or an inertial measurement unit.

As shown in FIG. 10, a first directional indicator 60 may be positionedon or within the movable object 10 and in communication with controller22. Directional indicator 60 may be configured to generate a signalindicative of a reference direction (e.g., a compass heading or otherreference directions) and communicate the reference direction tocontroller 22. Controller 22 may be configured to determine that thereference direction is the direction of the first axis of the localcoordinate system and store the direction within memory for furtherprocessing.

Controller 22 may be further configured to receive a second directionalsignal indicative of the direction of the first axis of the usercoordinate system. The controller 22 may receive the second directionalsignal from a second directional indicator, such as a compass, apositioning device, or an inertial measurement unit. As also shown inFIG. 10, a second directional indicator 62 may be positioned at anylocation where a user may be able to be positioned or controllingmovement of the movable object. For example, second directionalindicator 62 may be located on or within terminal 32 and be inelectronic communication with controller 22 (e.g., via communicationsystem 20). Second directional indicator 62 may be configured togenerate a signal indicative of a second reference direction (e.g., acompass heading or other reference directions) and communicate thesecond reference direction to controller 22. Controller 22 may beconfigured to determine that the reference direction is the direction ofthe first axis of the user coordinate system and store the directionwithin memory for further processing.

Controller 22 may be configured to perform a mathematical comparison ofthe first reference heading indicated by first directional indicator 60from the second reference heading indicated by second directionalindicator 62 to determine the offset between the local coordinate systemof the movable object 10 and the user coordinate system. In this way,the controller may be able to determine a directional (e.g., an angular)offset between a command generated in the user's perspective (i.e., withrespect to the user coordinate system) and a resultant movement of themovable object 10 in the user's perspective based on the command. Thisoffset may then be applied to commands generated in the user'sperspective using a perspective transformation to obtain a correspondingcommand in the perspective of the movable object 10.

Similar mechanisms may be included in movable object 10 and terminal 32to identify two other axes of the respective coordinate systems andallow controller 22 to determine the difference or offset for eachcorresponding pair of axes.

Once the 3-dimensional offset is determined, controller 22 may generatea transformation matrix to represent such offset, as follows:

$\begin{matrix}{T = {\begin{matrix}\alpha_{{xx}\;\prime} & \beta_{{xy}\;\prime} & \gamma_{{xz}\;\prime} \\\alpha_{{yx}\;\prime} & \beta_{{yy}\;\prime} & \gamma_{yz\prime} \\\alpha_{{zx}\;\prime} & \beta_{{zy}\;\prime} & \gamma_{{zz}\;\prime}\end{matrix}}} & \lbrack 1\rbrack\end{matrix}$

Controller 22 would then convert a user input r on terminal 32 wouldthen be converted through a matrix transformation to generate a signal sin the perspective of movable object 10:S=T·r  [2]where r and T are both 3-dimensional signals.

The translation or transformation described above may take into accountthe nature of the signal or command a user inputs on terminal 32, i.e.,whether the user desires for movable object 10 to make a translationalor rotational movement.

FIG. 11 shows another exemplary embodiment for determining the offsetbetween the perspective of the movable object 10 and the user'sperspective. In the example shown in FIG. 11, controller 22 may beconfigured to determine the offset between the perspective of themovable object 10 and the user perspective based on input from apositioning system 64. Controller 22 may be in communication with apositioning system 64 (e.g., via communication system 20) and configuredto track the location of movable object 10 and a reference point of theuser perspective (e.g., terminal 32) with respect to a referencecoordinate system. That is, the reference point of the user'sperspective and movable object 10 may each include a positioning device66 configured to receive and/or generate positioning signals usable bythe locating system 64 to determine the location and/or movements ofmovable object 10 and the reference point of the user perspective (e.g.,terminal 32).

The positioning system 64 may be configured to communicate the locationsof movable object 10 and the reference point of the user perspective(e.g., terminal 32) to controller 22. Using this information, controller22 may carry out an orientation process in which flight parameters(e.g., command signals for roll, pitch, and yaw) of movable object 10are tested to determine how movable object 10 responds directionallywith respect to the reference coordinate system to commands received inthe perspective of movable object 10 (i.e., in the local coordinatesystem). The results of the orientation sequence may allow controller 22to determine the direction of at least one axis of the local coordinatesystem with respect to the reference coordinate system. The direction ofa first axis of the user coordinate system may be assumed or initiallyaligned with tracking components of the positioning system 64 to allowfor the direction of a first axis of the user coordinate system to beunderstood by controller 22. Controller 22 may then be able to determinethe difference (e.g., an angular difference) between the localcoordinate system and the user coordinate system in the same manner asdescribed above as described above.

Another exemplary method of determining the direction of at least oneaxis of the local coordinate system of movable object 10 and the usercoordinate system for determining an offset between the two coordinatesystems includes initially aligning the two coordinate systems andtracking their subsequent movements in a single coordinate system. Forexample, at the beginning of a flight operation, the user may align hisor her perspective with a known direction of at least one axis of thelocal coordinate system of the movable object 10. For instance, the usermay stand in front of, behind, or adjacent to the movable object 10before the beginning of flight to orient himself or herself to the knownaxis direction of the movable object. The user may then provide an inputindicative of an alignment calibration (e.g., button 42) or simply beginoperating movable object 10. At this time, controller 22 may begin totrack the movements of movable object in each direction of the localcoordinate system during flight based on a directional indicator, suchas an inertial measurement unit. Using the signals from the directionalindicator, the controller may able to determine the angular displacementof at least one axis of the local coordinate system with respect to itsinitial position at the beginning of flight. This difference may beindicative of a current direction of the at least one axis of the localcoordinate system. In this way, the controller may be configured todetermine that the difference between the initial direction and thecurrent direction is the offset of between the local coordinate systemand the user coordinate system, assuming that the user coordinate systemis still aligned with the initial direction at the beginning of flightor at calibration.

In addition or alternatively, embodiments of the present disclosure mayprovide for user inputs not necessarily with respect to the multi-axisuser coordinate system, but rather natural commands, such as for movableobject 10 turn left or right from the user's perspective, regardless inwhich direction movable object 10 is headed. Controller 22 may beconfigured to translate such user commands into signals into theperspective of movable object 10, e.g., its local coordinate system, toeffect a change in flight behavior corresponding to the user's demand.For example, a user input desiring a greater height may be translatedinto a pitch command for the head of movable object 10 to turn up.Often, the user input would be translated into some combination oflinear and/or rotational movements, such as pitch, yaw, speed change,etc., as in a case of target tracking.

Target tracking is useful in many situations, such as surveillance andfilm shooting or video shooting, where a camera loaded on movable object10 is shooting a moving target, and movable object 10 must able tofollow the target and sometimes move around the target to give thecamera different view angles. Particularly in professional filmographyand videography, target tracking offers the advantage that a user mayonly need to be concerned with controlling the flight of movable object10, while movable object 10 automatically follows the target and makessure the target stays in sight for the filming, leading to more preciseflight control and consequently higher quality of imaging results (e.g.,photos and videos).

Controller 22 may be configured to utilize the input from sensing system18 to identify and/or track targets during flight. Tracking a targetduring flight may include identifying a target and maintaining thetarget in a field of sight of the movable object even while the targetand/or movable object 10 is moving. Maintaining the target within thetracking view of movable object 10 may require movable object 10 toautomatically adjust its flight parameters to keep sight of the target(i.e., in the perspective of movable object 10 or payload 14) whilereconciling these tracking movements with the desired movement ofmovable object 10 commanded by the user (i.e., from the user'sperspective or another reference perspective).

For example, with reference to FIG. 12, during target tracking, a usermay position movable object 10 to achieve desired perception of thetarget and subsequently maintain that perception using flight tracking.To position movable object 10 to the desired perception, the user mayinitially control the flight of movable object 10 to a desirableposition with respect to the target and then command movable object 10to start tracking the target. Movable object 10 would then automaticallyadjust its flight parameters to maintain the target in view. Flightparameters to achieve and maintain target tracking may include relativeposition (e.g., linear, angular, etc.), speed (e.g., linear, angular,etc.), and acceleration parameters (e.g., linear, angular, etc.) of themovable object with respect to the target. In some embodiments, thedesired flight parameters may be user specified. For instance,controller 22 may be configured to receive one or more input signalsfrom the user via terminal 32, which signals may be indicative of arelative height and distance of movable object 10 to maintain withrespect to the target during flight tracking. In other embodiments, theuser may, as mentioned above, simply position the movable object so asto give movable object 10 a desired perception of the target, andprovide an input indicative of a desire to track the target from thatperception.

Inputs received by controller 22 from the user relating to targettracking may be received from the user's perspective and may then betranslated to the perspective of movable object 10, in accordance withthe exemplary processes discussed above, for generating control signalsthat cause movable object 10 to track the target from the desiredposition and with the desired movement characteristics (e.g., speed,acceleration, etc.) desired by the user. While movable object 10 istracking the target, the user may wish to change one or more trackingparameters, such as the relative distance, speed, or accelerationbetween movable object 10 and the target, and generate commands viaterminal 32 to cause such changes from the user's perspective. Suchcommands may correspond to changes in the perspective of movable object10 or payload 14 (e.g., carrier 16, sensory devices 19, etc.) desired bythe user. Controller 22 may be configured to receive, from the user'sperspective, the commands indicative of the desired changes to theperspective of movable object 10 or payload 14, and translate thecommands to the perspective of movable object 10, in accordance with theexemplary processes discussed above, for generating control signals thatcause movable object 10 to track the target from the perspective desiredby the user.

In some embodiments, controller 22 may be configured to receive an inputsignal from an input device, such as button 42 of terminal 32, as anindication to begin tracking at the current perception of the movabledevice. In other embodiments, target tracking may be automaticallyinitiated by controller 22 based on other factors, such as being poweredup, an elapsed period of operating time, indication of the presence of atrackable target, or any other suitable input.

During target tracking, controller 22 may be configured to makenecessary adjustments to flight parameters (e.g., roll, pitch, yaw,throttle, etc.) in order to track the target, i.e., maintain the movableobject's perception of the target. A perception of a target may refer toa characterization of a target by one or more perceived parameterscorresponding to one or more determinable relative parameters betweenthe movable object and the target. Determinable relative parameters mayinclude, for example, relative positional and rotational parametersbetween movable object 10 and a reference point or reference object,such as the target, a ground surface, or another object. Relativepositional parameters and rotational parameters may include a relativedistance between movable object 10 and the target (e.g., a lineardistance, a radial distance, an angular distance, a vertical distance, ahorizontal distance, etc.) and a relative speed between the movableobject 10 and the target (e.g., a linear velocity, an axial velocity, aradial velocity, a tangential velocity, and angular velocity, etc.).

Perceived parameters for characterizing a perception may vary dependingon what method is used to perceive a target. For example, as shown inFIG. 13, controller 22 may be configured to receive signals indicativeof perceived parameters from a positioning system 64. From thepositioning system, controller 22 may be configured to constantlyreceive signals indicative of the location of the target and of movableobject 10. Controller 22 may be configured to constantly compare theperceived position of the movable object 10 to the perceived position ofthe target to constantly determine, for example, the relative distancebetween the movable object 10 and the target, the relative speed betweenthe movable object, and/or other relative parameters.

In other embodiments, perceived parameter for characterizing theperception of a target may be determined using a computer visionssystem. For example, as shown in FIG. 14A, movable object may include animaging device 68 supported by a carrier 16. The imaging device 68 maybe configured to generate optical data of the target for identifying andtracking the target. For example, the imaging device 18 may be anoptical device, such as a camera or video camera. The imaging device 68may be configured to generate imaging data indicative of one or morefeatures of the target, such as a height or width of the target.Controller 22 may include or be in communication with an image analyzer70 configured to analyze iterations of imaging data to identify featuresof the target that may be used to determine relative parameters betweenthe movable object 10 and the target. For example, image analyzer 70 maybe configured to establish a bounding box 72 around the target andmonitor how aspects of bounding box 72 changes. For example, as shown inFIG. 14A, bounding box 72 may begin near a center of an image window 74in a first imaging iteration. In a second imaging iteration, as shown inFIG. 14B, image analyzer 70 may determine that the position of boundingbox 72 in the image window 74 has moved by an amount ΔP_(x) in ahorizontal direction and an amount ΔP_(y) in a vertical direction. Basedon this information, image analyzer 70 may be configured to determinethat a horizontal distance between the movable object 10 and the targethas changed and in what direction. Further, the image analyzer 70 may beconfigured to observe changes in the height H and width W of thebounding box 72 to determine whether a relative height between themovable object 10 and the target changed.

As shown in FIG. 15A, image analyzer 70 may be configured to detectchanges in the width W and height H of bounding box 72 and determinechanges in the relative distance between the movable object 10 and thetarget. For example, as shown in FIG. 15B, as the movable object 10 getscloser to the target, the size of the bounding box 72 may increase inwidth W and height H. The image analyzer 70 may also be configured todetermine the rate of change of feature of the bounding box 72 todetermine relative speeds between the movable object 10 and the target.

FIGS. 15A and 15B show exemplary perceptions of the target from theperspective of the movable object 10. That is, bounding box 72 at aparticular location within the image window 74 and having particulardimensions (e.g., width W and height H) may be indicative of a currentperception of the target from the perspective of movable object 10. Thecurrent perspective of the target may also include relative linear,radial, tangential, and angular speeds of the movable object withrespect to the target as determined by changes in bounding boxparameters with respect to time. When the perception of the target fromthe perspective of the movable object 10 does not change, image analyzer70 may be configured to determine that all relative parameters betweenthe movable object 10 and the target are constant, and therefore thatmovable object 10 is currently tracking the target at the desiredtracking parameters.

To maintain the current perception of the target from the perspective ofmovable object 10, controller 22 may be configured to automaticallygenerate one or more control signals for adjusting flight parameters(e.g., roll, pitch, yaw, throttle, etc.) when changes in perception aredetected. In one example, controller 22 may be configured to generateand directly apply control signals to cause the desired flightparameters to be achieved. In this example, controller 22 may simplygenerate commands that will cause movable object 10 to expedientlyachieve the desired perspective. This method may be referred to as a“brute force” method. In this situation, a user input indicating alonger tracking distance between movable object 10 and the target may,for example, be translated to flight controls signals for movable objectto fly away from the target for a period of time and then restored toits previous flight pattern once the desired distance is achieved.

In another example, as shown in FIG. 16, controller 22 may be configuredto use feedback control of flight parameters to maintain the currentperception of the target from the perspective of movable object 10 whileproviding for smoother movement and continued tracking during flightparameter adjustment. For example, image analyzer 70 may receive imagingdata from imaging device 68 and determine a current relative distance D₁between the movable object 10 and the target. Controller 22 may comparethe current relative distance D₁ to an initial relative distance D₀,which may be a desired relative distance or a relative distance from aprevious iteration. A feedback controller 76 may be configured todetermine whether there is a difference between the initial relativedistance Do and the current relative distance D₁. When the two are thesame, movable object 10 is following at the correct distance and nochange in flight parameters is necessary. When there is a difference,feedback controller 76 may be configured to determine a control speedV_(Cy) for movable object 10 in the direction of the relative distancefor adjusting the current relative distance to match the initialrelative distance.

Feedback controller 76 may be configured to carry out feedback controlusing any suitable feedback process. For example, feedback controllermay be configured to carry out proportional control (P), integralcontrol (I), derivative control (D), proportional-integral control (PI),or proportional-integral-derivative control (PID). Feedback controller76 may also or alternatively be configured to apply one or moremathematical filters to conduct or enhance feedback control processes.It is understood that other or additional feedback control and filteringprocesses may be used.

An actuation system 78 may be configured to generate motion controlsignals to one or more actuators for achieving the control velocitydetermined by feedback controller 76. The control signals generated byactuation system 78 may control a flight parameter, such as pitch, inorder to cause the movable object to move closer or farther from thetarget to achieve the initial relative distance. This process may repeata number of times until the initial or desired distance is achieved.

It is noted that the process shown in FIG. 16 and described above may beperformed for other tracking control parameters in addition to therelative distance D₀, such as a relative height, a relative radialdistance, a relative axial distance, a relative angular displacement, arelative linear speed, a relative radial speed, a relative tangentialspeed, a relative vertical speed, etc. Whenever a difference is detectedin one of the tracking control parameters, either between an initialvalue and the current value or between a desired value and the currentvalue, the feedback process shown in FIG. 16 determines appropriateflight parameters for movable object 10, such as translational and/orrotational speed in the correct direction or about the correct axis, forreducing the difference between the initial and current parameters, andprovides corresponding motion control signals to movable object 10 toachieve any necessary adjustment in the flight parameters. For instance,adjustments to roll, pitch, yaw, and throttle may be made to achievedetermined control speeds along (i.e., translationally) or about (i.e.,rotationally) any axis of movement of movable object 10.

During a photo or video shoot, the user may change the positionalrelationship with movable object 10 as well as the camera or otherimaging equipment loaded thereon, and also possibly the positionalrelationship with the target, either because the user wants to change aperception of the shot or because of the movements of the target andmovable object 10. Thus, the user may wish to have a controlledtransition from a first perception to a subsequent perception and, oncethe movable object arrives at the subsequent perception, controlledmaintenance of the subsequent perception to continue producing highquality imaging results.

As one example, the user may wish to change the perception by closing inon the perception of the target, i.e., shortening the distance betweenmovable object 10 and the target. To do so, the user may simply actuatean input device, such as first or second input lever 38, 40 of terminal32 to generate a command signal. Controller 22 may receive this userinput and respond by adjusting one or more flight parameterscorresponding to the user input to achieve the desired change indistance. But, the feedback control process shown in FIG. 16 is designedto maintain the initial or desired distance D₀ (or a desired perceptionof the target in the view of movable object 10). Any attempt to adjustthe distance by simply changing the flight speed of movable object 10would be negated by the works of the feedback control process.

Consistent with embodiments of the present disclosure, controller 22 maybe configured to determine a subsequent or future perception of thetarget from the perspective of movable object 10 based on the user'scommand to change a tracking parameter (e.g., the following distance),thereby adjusting the initial or desired tracking distance Do to allowthe feedback control process to work in cooperation with the user tocontrol movable object 10. For example, as shown in FIG. 17, controller22 may be configured to perform a feedback control method similar to themethod described above in connection with FIG. 16 to determine asubsequent or future perception of the target from the perspective ofthe movable object 10 using an initial position estimator 80. Initialposition estimator 80 may be configured to receive data from the imageanalyzer 70 as well as the input from the user to determine what thefuture perception of the target from the perspective of the movableobject 10 will be based on this information. In the example shown inFIG. 17, this process results in a modified initial or desired distanceD₀, after which the feedback control process will try to maintain thatmodified distance D₀ between movable object 10 and the target.

With reference to FIGS. 18 and 19, the user input may be a signalgenerated by an input device, such as first or second input lever 38, 40of terminal 32. The generated signal may be correlated to changes inperception of the target, such as changes in size (e.g., width W andheight H) of bounding box 72 and resulting corresponding changes inflight parameters (e.g., roll, pitch, yaw, throttle, etc.). In this way,initial position estimator 80 (FIG. 17) may be configured to determine asubsequent perception of the target that corresponds to a subsequent ordesired difference in one or more relative parameters between themovable object and the target (e.g., a relative linear distance, arelative angular displacement, a relative height, a relative linearspeed, a relative angular speed, a relative radial speed, a relativevertical speed, etc.).

With reference to FIG. 17, the subsequent relative parameter determinedby initial position estimator 80 may then be compared to thecorresponding current relative parameter (e.g., based on the currentperception of the target), and the difference may be used in feedbackcontrol. In this way, a difference between the current perception andthe subsequent perception may yield a difference between a subsequentrelative parameter and a current relative parameter, and this differencein the relative parameters may be used to generate the control speedV_(Cy) for moving movable object 10 to maintain the target (i.e., thebounding box 72) within image window 74 (referring to FIGS. 18 and 19)of image analyzer 70.

In some embodiments, the determined control speed V_(Cy) may be fed backto the initial position estimator 80, and, during a subsequent iterationof the feedback control process, the initial position estimator 80 maybe configured to re-determine the subsequent perception of the targetbased on the control speed to more accurately determine the subsequentperception of the target. For example, feedback controller 76 maydetermine a control speed for allowing relative parameters between themovable object 10 and the target to be changed based on the user's inputcommand while simultaneously preventing the user's input command frommoving the bounding box in the image window (referring to FIGS. 18 and19) to prevent the tracking process from being stopped. Thus, thedetermined control speed may be a more realistic speed with which themovable object 10 will respond to the user's input. Accordingly, a moreaccurate estimation of the subsequent perception of the target mayaccount for the fact that the subsequent perception will not actually beas different from the current perception as it would otherwise be ifonly the user input, and not the control speed, dictated the response ofthe movable object.

In some situations, the accuracy of perception and perception estimationcan be further increased by translating perception information receivedsensory devices, such as imaging device 68, from the perspective of thesensory device to the perspective of the movable object 10. For exampleduring some maneuvers, such as when circling the target, tracking in anarcuate path, or tracking the target while making somewhat sharpmaneuvers (e.g., high-speed turns, steep banking turns, low radiusturns, etc.), the ability of the carrier 16 to quickly adjust thepayload 14 (i.e., the imaging device 68) for tracking the target canresult in situations where the perception of the target from theperspective of the imaging device is slightly different from theperception of the target from the perspective of the movable object 10.When such is the case, the subsequent perceptions determined by initialposition estimator 80 may be slightly inaccurate, which can cause thefeedback controller 76 to determine slightly inaccurate control speedsfor controlling the movable object 10 to during target tracking. Toaccount for any offset in perspective between the imaging device 68 orcarrier 16 and the movable object 10, the controller may be configuredto receive perception information from the perspective of the imagingdevice 68 or carrier 16 and transform the perception information fromthe perspective of the imaging device 68 or carrier 16 to theperspective of the mobile object 10. In this way, subsequentperspectives and commands for controlling flight parameters may moreaccurately reflect the true perspective of the movable object 10. Toperform the perspective transformation, the controller of the movableobject 10 may be configured to constructing a matrix representation ofthe sensory input in terms of a coordinate system associated with theimaging device 68 or carrier 16, and transforming the matrix into amatrix representation of the sensory input in terms of the localcoordinate system of the movable object based on an offset between theperspective of the imaging device 68 or carrier and the perspective ofthe movable object 10. Techniques for determining the perspective offsetas described above may be used.

Sometimes movable object 10 moves around the target, for example, incircles, to provide the camera different view angles. FIG. 20illustrates a situation where movable object 10 moves around a target ina circle with radius r, with a radial velocity component V_(RAD), atangential velocity component V_(TAN), and an angular velocity ω. Thetracking control process described above in connection with FIGS. 16 and17 may be employed to maintain or adjust the radius r. The user may wishto have movable object 10 move faster or slower for varying effects ofthe imaging results. Centrifugal forces associated with the circularmovement, which pull movable object 10 outwardly in the radialdirection, may be countered by the inward flight of movable object 10,e.g., via a pitch control. But the pitch amount, or a tilt angle, ofmovable object 10 has a physical limit, often because of the limit ofhow much carrier or gimbal 16 can tilt. If the tangential velocityV_(TAN) exceeds a threshold V_(Tmax), the pitch of movable object 10will not be able to counter the centrifugal forces, which may ultimatelythrow movable object 10 off the circular track and result in a loss oftrack of the target.

To prevent users from exceeding the maximum allowable tangential speedV_(Tmax), the controller of movable object 10 may be configured todetermine the maximum allowable tangential velocity for sustainingtarget tracking and limit the ability of the user to increase thetangential velocity of the movable object 10 while tracking a target ina circular pattern. For example, controller 22 may be configured tofirst determine the radius r of rotation of movable object 10 about thetarget. One method for determining the radius r may include dividing anarc length L of travel by an angular displacement θ along the arclength, i.e.,

$r = {\left( \frac{L}{\theta} \right).}$The arc length L of travel can be determined in a number of ways. Forexample, controller 22 may be configured to integrate the tangentialvelocity of movable object 10 over time to calculate the arc length L.And as discussed above, the tangential velocity of movable object 10 canbe determined by analyzing image data collected by imaging device 68,such as perceived changes in the size, shape, and location of the targetfrom the perspective of the imaging device 68. In other embodiments,data from sensory devices, such as a GPS system or an inertial measuringunit (IMU) may be collected and analyzed to determine the arc length Ltraveled by movable object 10 over a period of time. The angulardisplacement θ of the movable object 10 over the arc length L may bedetermined by analyzing image data collected by imaging device 68, suchas perceived changes in the size, shape, and location of the target fromthe perspective of the imaging device 68 (i.e., an optical sensorydevice), as discussed above, or from infrared imaging devices,ultraviolet imaging devices, x-ray devices, ultrasonic imaging devices,radar devices, etc.

Another method for determining the radius r may be dividing thetangential velocity V_(TAN) by the angular velocity ω of movable object10, i.e.,

$\left( \frac{V_{T}}{\omega} \right).$As discussed above, both the tangential velocity and the angularvelocity of movable object 10 may be determined by comparing multipleperceptions of the target from the perspective of the movable object,the multiple perceptions being generated with data collected by imagingdevice 68 or other devices, such as infrared imaging devices,ultraviolet imaging devices, x-ray devices, ultrasonic imaging devices,radar devices, etc. Changes in the size, shape, and location of thetarget (i.e., the bounding box 72) determined during the comparison maycorrespond to values of tangential velocity and angular velocity, whichmay be used to determine the radius of rotation of movable object 10.

Another method of determining the radius r is described with referenceto FIG. 21. As shown in FIG. 21, the movable object 10 with carrier 16may carry an imaging device, such as a camera 82. Camera 82 can capturean image of the target on an image plane 84 with an image coordinates86. Using similar techniques as described above, a bounding box 72 mayalso be generated for analyzing geometry shapes and providing continuousand stable results.

As shown in FIG. 21, the image of a target may be represented based onan aperture imaging model, which assumes that a light ray from an objectpoint in a three dimensional space can be projected on an image plane toform an image point. The optical axis 88 can pass through both themirror center 90 and the image center. The distance between the mirrorcenter and the image center can be equal or substantial similar to thefocal length 92. For illustration purposes only, the image plane 84 canbe moved to the mirror position on the optical axis 88 between themirror center 90 and the target.

In accordance with various embodiments of the present disclosure, thesystem can perform an initialization step, which includes estimatingboth the target distance and target height. Here, the system candetermine the projected relative distance 94 on the ground between themirror center 90 and the target based on the geometry relationship.Then, the system can determine the target height.

At the initialization stage (i.e., when t=0), the system can assume thatthe altitude of the movable object 10 (i.e. the camera 82) is measuredfrom the same floor (or horizontal line) where the target stands.Without limitation, when the floor is not a horizontal line, the systemcan use the effective altitude, which accounts for the altitudedifference, to replace the actually measured altitude of the movableobject 10 for measuring the target distance and target height.

In the example as shown in FIG. 21, the target may have a top targetpoint (x_(t), y_(t), z_(t)) and a bottom target point (x_(b), y_(b),z_(b)) in a world coordinates, which are projected on the image plane 84as a top image point (u_(t), v_(t)) and a bottom image point (u_(b),v_(b)) respectively in the target image 96. A top line passes throughthe mirror center 90, the top image point, and the top target point andcan have a tilt angle 98 from the axis Z of the world coordinates. Also,a bottom line passes through the mirror center 90, the bottom imagepoint, and the bottom target point and can have a tilt angle 100 fromthe axis Z.

Thus, the target top and bottom direction vectors {right arrow over (T)}and {right arrow over (B)} can be expressed as in the following.

$\overset{\rightarrow}{T} = {\begin{pmatrix}x_{t} \\y_{t} \\z_{t}\end{pmatrix} \sim {{RK}^{- 1}\begin{pmatrix}u_{t} \\v_{t} \\1\end{pmatrix}}}$ $\overset{\rightarrow}{B} = {\begin{pmatrix}x_{b} \\y_{b} \\z_{b}\end{pmatrix} \sim {{RK}^{- 1}\begin{pmatrix}u_{b} \\v_{b} \\1\end{pmatrix}}}$where K represents the intrinsic matrix of the camera, and R representsthe camera rotation.

Then, the system can estimate the target distance based on the measuredor effective camera altitude 102 and the position of the bounding box104 in the image coordinates 86. For example, the distance 106 can becalculated as d=−h_(c)/z_(b)*P_(b), and the target height 108 can becalculated as h_(o)=h_(c)+z_(t)d/P_(t), where P_(b) is the projectionlength of {right arrow over (B)} on the ground and P_(t) is theprojection length of {right arrow over (T)} on the ground, which aredefined as in the following.P _(b)=√{square root over (x _(b) ² +y _(b) ²)}P _(t)=√{square root over (x _(t) ² +y _(t) ²)}

After the initialization step, the system can estimate the distance 106from the target, even when the target altitude changes (e.g. when thetarget is off the ground) and when the altitude of the movable object 10(e.g. a UAV) is unknown. This is beneficial, since during the tracking,the object may climb up or go down and the altitude of the UAV may beunreliable as the UAV flies over grasses or climbs up (e.g. 5 metersabove the ground).

As shown in FIG. 21, after the initialization, the projected relativedistance 106 on the ground between the target 10 and the movable object10 can be calculated as h_(c)/dh, where dh present the estimate heightof the target at a unit distance away from the camera, which can becomputed using the following formula.

${dh} = {{\frac{\overset{\rightarrow}{T}}{P_{t}} - \frac{\overset{\rightarrow}{B}}{P_{b}}}}$

Thus, the method can be efficient and may have very few restrictionsonce it is initialized, since the system can estimate the distance 106from the object based on the height of the target after theinitialization.

After determining the radius r of rotation, the maximum tangentialvelocity V_(Tmax) may be determined through the relationship with amaximum centripetal acceleration

$a_{\max} = {\frac{V_{t\mspace{11mu}\max}^{2}}{r}.}$The maximum centripetal acceleration of movable object 10 may beinherently determined by its structural design, in that, regardless ofother flight parameters, movable object 10 may reach a certain maximumtilting angle at which it can no longer sustain flight and will stall.In other words, as movable object 10 is tilted to increase itscentripetal acceleration, a maximum acceleration will be reached whenmovable object 10 reaches a tilting angle at which it can no longersustain controlled flight. This maximum angle may be determinedempirically or may be provided by the manufacturer of movable object 10.Thus, assuming a known value for a_(max), the maximum tangentialvelocity V_(Tmax) can then be determined.

When movable object 10 is circling the target, the controller of movableobject 10 may be configured to receive a user input (e.g., via one ormore input levers 38, 40 of terminal 32) indicative of a selection of adesired tangential velocity between a maximum desired tangentialvelocity and a minimum desired tangential velocity. The controller ofmovable object 10 may be configured to receive the user input via one ormore of a lever, stick, button, dial, knob, touch screen, or electronicdevice. In some embodiments, the electronic device may include memoryand at least one processor configured to execute a program for providingan “app” or one or more graphical user interfaces for receiving inputfrom the user for selecting a desired tangential velocity.

As used herein, the term “between,” when used with reference to a rangeof values, may be inclusive of endpoint values. That is, the user mayprovide input indicative of, for example, a roll setting that canincrease or decrease the tangential velocity of the movable object whilecircling the target. When the user moves the input device for adjustingthe roll setting all the way to a maximum displacement, a signal for amaximum roll setting may be generated. However, applying maximum roll tothe movable object may cause it to stall during flight, and so thecontroller 22 may be configured to limit the user's input and onlypermit roll settings that will ensure the movable object will remainbelow its maximum tangential velocity determined for the current radiusof rotation. That is, the controller of movable object 10 may set themaximum desired tangential velocity that is possible to select to alimit value that is no greater than the maximum allowable tangentialvelocity of the movable object 10. In this way, the user may beprevented from applying too much roll and stalling the movable object10. In some embodiments, the maximum desired tangential velocity that ispossible to select may be further limited, such as to a value below themaximum tangential velocity or by a percent of the maximum tangentialvelocity (e.g., 95%, 90%, 85%, etc.). In this way, it may be less likelythat the movable object may be caused to reach a maximum roll value andbe suddenly unable to maintain controlled flight.

In some situations, such as when movable device is circling and trackinga target, the user may wish to increase the radius r of rotation of themovable object or gradually circle away from the target and terminatethe tracking operation, as shown in FIG. 22. To allow the user toperform such a maneuver, the user controller of movable object 10 may beconfigured to set the maximum desired tangential velocity that can beselected to a limit value above the maximum allowable tangentialvelocity of the movable object. For instance, the maximum desiredtangential velocity that can be selected may be set to a value orpercentage that will allow the user to gradually increase the radius ofrotation of movable object 10 by increasing the tangential velocity andovercoming centripetal forces acting on movable object 10. In this way,the controller may be configured to allow the movable object 10 to trackthe target at a larger radius of rotation or to gradually circle awayfrom the target and terminate the tracking operation.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed methods andsystems. Other embodiments will be apparent to those skilled in the artfrom consideration of the specification and practice of the disclosedmethods and systems. It is intended that the specification and examplesbe considered as exemplary only, with a true scope being indicated bythe following claims and their equivalents.

What is claimed is:
 1. A method for controlling a movable object in asystem that includes one or more propulsion devices and a controller incommunication with the one or more propulsion devices, comprising:receiving, through the controller, a user input indicative of a commandto adjust a perception of a target while tracking the target, the userinput including a first parameter corresponding to a first coordinatesystem; converting, through the controller, the user input including thefirst parameter to the user input including a second parametercorresponding to a second coordinate system, including: determining atleast a first axis of the first coordinate system through a first signalindicative of the first axis of the first coordinate system; determiningat least a first axis of the second coordinate system through a secondsignal indicative of the first axis of the second coordinate system; anddetermining an offset between the first axis of the first coordinatesystem and the first axis of the second coordinate system, andconverting the user input including the first parameter to the userinput including the second parameter based on the offset; determining,through the controller, a subsequent perception of the target based onthe user input including the second parameter; generating, through thecontroller, one or more control signals based on the subsequentperception of the target; and controlling, using the one or more controlsignals, the one or more propulsion devices to move the movable object.2. The method of claim 1, wherein the subsequent perception of thetarget corresponds to a relative displacement between the movable objectand the target.
 3. The method of claim 2, wherein the relativedisplacement between the movable object and the target includes at leastone of a horizontal displacement or a radial displacement.
 4. The methodof claim 2, wherein the relative displacement includes at least one of avertical displacement, a height, or an axial displacement.
 5. The methodof claim 1, further comprising: determining a control speed for movingthe movable object.
 6. The method of claim 1, wherein determining thesubsequent perception of the target includes receiving input from asensory device.
 7. The method of claim 1, further comprising:determining a radius of rotation of the movable object about the target.8. The method of claim 7, wherein the radius of rotation is determinedbased on an arc length of travel and an angular displacement of themovable object.
 9. The method of claim 8, wherein the arc length isdetermined based on an integration of a tangential velocity of themovable object.
 10. The method of claim 8, wherein the arc length isdetermined based on input received from at least one of a positioningsystem or an inertial measurement unit.
 11. The method of claim 8,wherein the angular displacement is determined based on informationdetected by the movable object.
 12. The method of claim 11, wherein theinformation detected by the movable object is generated by a sensorydevice.
 13. The method of claim 12, wherein the sensory device includesan optical device.
 14. The method of claim 7, wherein the radius ofrotation is determined based on a tangential velocity of the movableobject and an angular velocity of the movable object.
 15. An unmannedaerial vehicle (UAV) system comprising: one or more propulsion devices;and a controller in communication with the one or more propulsiondevices and configured to control the UAV to track a target, thecontroller including one or more processors configured to: receive auser input indicative of a command to adjust a perception of a targetwhile tracking the target, the user input including a first parametercorresponding to a first coordinate system; convert the user inputincluding the first parameter to the user input including a secondparameter corresponding to a second coordinate system, including:determining at least a first axis of the first coordinate system througha first signal indicative of the first axis of the first coordinatesystem; determining at least a first axis of the second coordinatesystem through a second signal indicative of the first axis of thesecond coordinate system; and determining an offset between the firstaxis of the first coordinate system and the first axis of the secondcoordinate system, and convert the user input including the firstparameter to the user input including the second parameter based on theoffset; determine a subsequent perception of the target based on theuser input including the second parameter; generate one or more controlsignals based on the subsequent perception of the target; and control,using the one or more control signals, the one or more propulsiondevices to move the UAV.
 16. The UAV of claim 15, wherein the subsequentperception of the target corresponds to a relative displacement betweenthe UAV and the target.
 17. The UAV of claim 15, wherein the one or moreprocessors are further configured to: determine a control speed formoving the UAV.
 18. The UAV of claim 15, further comprising: a sensorydevice configured to provide input to the one or more processors fordetermining the subsequent perception of the target.
 19. The UAV ofclaim 15, further comprising: a sensory device; wherein the one or moreprocessors are further configured to determine a radius of rotation ofthe UAV about the target based on an arc length of travel and an angulardisplacement of the UAV, the angular displacement of the UAV beingdetermined based on information generated by the sensory device.
 20. Anon-transitory computer readable medium storing instructions that, whenexecuted, cause a computer to perform a method for controlling a movableobject in a system that includes one or more propulsion devices and thecomputer in communication with the one or more propulsion devices, themethod comprising: receiving, through the computer, a user inputindicative of a command to adjust a perception of a target whiletracking the target, the user input including a first parametercorresponding to a first coordinate system; converting, through thecomputer, the user input including the first parameter to the user inputincluding a second parameter corresponding to a second coordinatesystem, including: determining at least a first axis of the firstcoordinate system through a first signal indicative of the first axis ofthe first coordinate system; determining at least a first axis of thesecond coordinate system through a second signal indicative of the firstaxis of the second coordinate system; and determining an offset betweenthe first axis of the first coordinate system and the first axis of thesecond coordinate system, and converting the user input including thefirst parameter to the user input including the second parameter basedon the offset; determining, through the computer, a subsequentperception of the target based on the user input including the secondparameter; generating, through the computer, one or more control signalsbased on the subsequent perception of the target; and controlling, usingthe one or more control signals, the one or more propulsion devices tomove the movable object.